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Abstract — We consider transmission over a general memory- 
less cliannel, with bounded decoding complexity per bit under 
message passing decoding. We show that the achievable rate is 
bounded below capacity if there is a finite success in the decoding 
in a specified number of operations per bit at the decoder for 
some codes on graphs. These codes include LDPC and LDGM 
codes. Good performance with low decoding complexity suggests 
strong local structures in the graphs of these codes, which are 
detrimental to the code rate asymptotically. The proof method 
leads to an interesting necessary condition on the code structures 
which could achieve capacity with bounded decoding complexity. 
We also show that if a code sequence achieves a rate e close to 
the channel capacity, the decoding complexity scales at least as 
0(log(i)). 

I. Introduction 

In [1] and [2], Khandekar and McEliece suggested studying 
the problem the problem of per bit decoding complexity for 
capacity achieving codes. The authors conjectured on some 
lower bounds for the asymptotic decoding complexity for 
a sequence of codes achieving a rate which is e close to 
the channel capacity. For general memoryless channels, with 
message passing decoding, the authors conjectured a lower 
bound of 0(i log (i) ) operations per bit at the decoder. There 
is one exception: for the BEC, codes were already known [3] 
which achieve capacity with O (log i) operations per bit at 
the decoder (see [4]). The BEC is an exception because along 
each edge in the graph, a message needs to be passed only 
once. 

Soon after, in [5] [6] the authors constructed codes 
which achieve capacity with bounded decoding complex- 
ity for the BEC. In [7], the authors construct systematic 
Accumulate-Repeat-Accumulate codes which achieve capac- 
ity with bounded decoding complexity. Interestingly, all the 
known codes which achieve capacity with bounded decoding 
complexity use accumulation. 

It is a natural question whether there exist codes which 
achieve capacity with bounded decoding complexity under 
message passing decoding for general memoryless channels. 
It is a hard question, too, for we yet do not know of any codes 
which achieve capacity under message passing decoding for 
general memoryless channels. 

In this paper, using an insight developed in [8], we show that 
a lower bound of 0(log(i)) holds for a large class of codes, 
including LDPC codes and LDGM codes, and any sequence 
of systematic codes. 



This paper is organized as follows. In Section |ll] we intro- 
duce the notation and some definitions used in this article. In 
Section |III] we put forward our lower bound on the decoding 
complexity. We also find some bounds on the rate of codes 
given their decoding performance. In Section |IV] we observe 
a necessary condition for codes to achieve capacity with 
bounded decoding complexity: for infinite block-length, there 
should be no decrease in probability of error in finite number 
of operations per bit at the decoder. Section [V] concludes the 
paper 

II. Notation and Definitions 

The block length of a code is denoted by n. A vector com- 
prising of first i elements of a sequence of random variables 
is denoted in bold letters with superscript i. For example, the 
channel input vector is denoted by X" :— {Xi, X2, . . . , Xn}- 
Similarly, the channel output vector is denoted by Y". /c*'* 
element of any vector X" is denoted by Xk ■ It would also be 
convenient to define Y^^^ as the outputs of the variable nodes 
in the neighborhood of node i till the Z*'' iteration (excluding 
node i). C is used to denote the capacity of the channel under 
consideration. We assume that the channel is binary input, 
symmetric, and memoryless, but is otherwise arbitrary. 

Let e be the gap from capacity, R ~ C — e. Denote the 
'success in decoding' by a parameter r. t may correspond 
to decrease in probability of error, or a similar parameter 
characterizing some success in decoding 

We denote the number of decoding operations required per 
information bit for decoding a sequence of codes by Xd \^)' 
where e is the difference between the code rate R and the 
channel capacity C. 

All the results in this paper hold for general memoryless 
channel holds with one exception, the BEC. 

III. An upper bound on the rate and a lower bound 
ON the decoding complexity 

Suppose we have a sequence of LDPC codes of fixed 
degree distributions {X,p)- In asymptotic (large block-length) 
analysis, the probability density of the messages passed [9] 
converges in distribution to Soo, a point mass at infinity, as 
the number of iterations / converges to infinity. A message 

'Later, we use the decrease in entropy of an output symbol from H{Yi) 
as the parameter See (T). 



to i*'* variable node is a log likelihood ratio 



Pr(Xi = l|Y<'') 
(On a 



Therefore, the random variable Pr(Xi — 0|Y^^) A 1, and 
i?(Xi|Y£)) as Z ^ cxD. The randomness is over the 
channel realization, the choice of the code in the ensemble, 
and the choice of the node. 

Given some success in decoding by the l*^ iteration, with 
high probability the conditional entropy of the i*'' output bit 
Yj can then be bounded as follows 



(1) 



This was first observed in [8] in the context of LT codes. 

For some finite decoding success in constant number of 
computations, we prove that the rate is bounded below channel 
capacity, by explicitly finding the bound. We then arrive at a 
lower bound on the decoding complexity for a code being used 
at rate R = C — e. For derivation of these bounds, we first 
need a bound on _ff (Y"). 

A. Bound on H{Y" ) 

Consider the decoding up to I iterations. Let e; denote the 
probability of error after / iterations. 

After / iterations, the conditional entropy H{Yi\Y'f^l) can 
be bounded above by 



HiY,\Y^l\)<H{Y,)-n 



(2) 



Using this bound, we wish to upper bound the entropy of 
vector Y". Using the chain rule 



iI(Y") - > i/(K,|Y*~^) 



i=l 



(3) 



Naturally enough, we would like to use the fact that condition- 
ing reduces entropy to upper bound the individual conditional 
entropies in the sum. To that end, we want to remove some 
elements of the vector Y*^^ in order to arrive at Y^|. But 
notice that this would not always be possible. In fact, it is 
not difficult to show that in general, no particular permutation 
of the vector Y can ensure that y£| C Y*^^ for a constant 
fraction of Yi's. 

In [8], this problem is dealt with by showing that the set 
Y^^\Y*~^ can be considered to be erased for the analysis, and 
they try to find another constant t' which bounds the entropy 
H{Y^\y'-J^\ n Y'-i). In the following, we show a neater way 
to bound H{Y) which directly leads to a clean upper bound 
on the rate. Moreover, this method works for LDPC codes as 
well as some other classes of codes, which is not the case for 
the the method in [8]. 

Consider all the nl permutations of Y". Consider the set 
y£^, constituted by the output values of the bits corresponding 
to the nodes in the neighborhood after I iterations. For each i, 
this set has a constant number of elements equalling ki (say). 
It is important to note that for a fixed sequence of codes, fc; is 

^Strictly speaking, in [8], tlie bound is on mutual information. But tliat can 
easily be translated to a bound on conditional entropies that we are dealing 
with. 



independent of n. Because this set is constant in size, we try 
to find the fraction of permutations for which this set occurs 
before the particular Yi we are considering. The problem is 
equivalent to finding the number of permutations of a set in 
which a particular element of a subset occurs last amongst all 
the elements of the subset. This would be true for exactly 
fraction of total permutations, since it is the same for each 
element of the subset. 

The size of the neig hborhood set ki ^J2]{a- l)'(/3 - 1)% 
where a {(3) is the average variable (check) node degree. Also, 
^ — \ — R, where R is the code rate. 

We now derive the bound. In the following the bracketed 
(j) represents the j*'* permutation ordering of the vector Y". 



-i/(Y") 

n 



n X n 
1 



n! 

^^i/(Y"(j)) 



n! n 



n X n 



5]5]i/(y.(j)|Y-i(j)) 



3=1 i=l 

Exchanging the two summations 



n n: 

-i7(Y")=-5:-5:i/(r.(,)iY-i(j)) 



n ^ — ' nl 

As argued in the previous paragraph, the terms in the summa- 
tion can be bounded above by H{Yi) — t for fraction of 
these permutations. Therefore, 

z— 1 

Since Y^'s are identically distributed, we get the bound 

-i/(Y") < H{Y,) - ^ (4) 
n ki 

Per se, this bound is very loose, since the fraction ^ converges 
to polynomially in the average degree. 

In comparison with [8], this bound is explicit on its depen- 
dance on the degree distribution. In the following section, we 
show how this bound leads to a bound on the achievable rate 
for LDPC codes. 



B. Bound on the achievable rate for given performance after 
a fixed number of iterations 

Using Fano's inequality, 

H{Pe) > -i/(X"|Y") (5) 

n 

Using standard information-theoretic equalities 

-i?(X"|Y") = ii7(X") - -i?(Y") + ii?(Y"|X") (6) 

n n n n 

For any code, X" is the encoded data which is in 1-1 mapping 
with the information symbols. The information symbols are 
uniformly distributed over the 2"^ values. Thus, X" takes 
any value from the 2"^ codewords with uniform probability 



2 



distribution. Therefore 



nR 



Using the fact that the channel is memoryless 

l^(Y"iX") - H{Y,\X,) = H{Y,) - C 



(7) 



(8) 



Notice that in it now suffices to upper bound the entropy 
H(Y"), which is what we found in (Hjl. Along with ^ we 
obtain 



H{P,)>R~{H{Y,] 

Therefore, for 
achievable rate 



^)+H{Y,)-C = R+^~C (9) 



0, we get the following bound on the 

Tl 



R<C - 



(10) 



Also, notice that (|9]l gives us a lower bound on the probability 
of error for the case when this bound is violated. 

1 ) Tightening the bound with more knowledge of the code 
performance: Since the bound is valid for all iterations I, we 
get the following tighter bound 



ii/(Y") < min \H{Yi) - ^ 



(11) 



where the minimum is taken over all the values of / for which 
the decoding performance is known. 



Then, 



< sup{(a + /?- 
i>i 

< sup{(a + /3f+^} 
i>i 



< 



since R < 1. Define 



kid — sup 




Then k^'^ < oo for each finite c because as I 



(c) , and as I 



oo. 



bd 



0. 



Thus a bound on the achievable rate is 

T-(c) 



R<C- 



Hd 



Since k^/) < cxi, i? is bounded below capacity. 



(15) 



1, k 



bd 



(16) 



D. A lower bound on decoding complexity as a function of 
gap from capacity 



Substituting i? = C - e in (O, 



Hd - 



T-(c) 



(17) 



C. Bound for given decoding success in fixed number of 
computations 



Denote the neighborhood size by k^"^^ (which depends on I 
and a), and the success in decoding by t'-'^K Then it is easy 
to see that the same derivation still works, and we get the 
following bound on entropy of Y" 



ii7(Y«)<W0-^ 



(12) 



The total number of operations at the decoder is {a x l+f3 xl)n 
Therefore, the number of operations per information bit, 

_ {a + 13)1 



R 

Define fcmax as the maximum value of k^'^'> for fixed c. 
fcmax = sup|^(a-l)X/3-l)4 



(13) 



(14) 



(c) 

Numerically, we observed that fc^^ < e'^ for c large enough. 
Therefore, for fixed r, the per bit decoding complexity 



xL"' > O ( log 



(18) 



Here r is the parameter which reflects the success in decoding 
in O. 



E. Generalization to other codes 

Notice that we did not use any structure of LDPC codes for 
deriving the bound above. All we require is some finite success 
in decoding the output bits in order to decode the information 
bits. The bounds above, therefore, generalize to LDGM codes. 

If we assume plain message passing decoding, then for 
any systematic code, there has to be a finite success in the 
decoding of the information bits with a finite neighborhood 
size. This implies a finite success in the decoding of output 
bits! Therefore, no systematic code can achieve capacity with 
bounded decoding complexity under plain message passing 
decoding for general memoryless channels. 
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IV. A NECESSARY CONDITION FOR CODES WHICH 
ACHIEVE CAPACITY WITH BOUNDED DECODING 
COMPLEXITY 

Note that the results in Section |III] are not vaUd for the 
BEC. It is shown in [5] [6] that there exist codes which 
achieve capacity with bounded decoding complexity for the 
BEC. Considering the above bound, it is intriguing that such 
codes exist. After all, the above bound only depends on 
some knowledge of marginal of output bits. As long as the 
probability of the output bits being 1 is bounded away from 
0.5, the bound holds. 

The codes in [5] are non-systematic IRA codes. IRA codes 
have an LDGM inner code, and a convolution code as the outer 
code. The convolution structure means that for any particular 
bit, for any finite number of iterations I, the marginal prob- 
ability Pr(yi = 1|Y^|) = 0.5. The marginal uncertainty of 
the output bits does not decrease with increase in I ! This also 
hints why systematic IRA codes, for which some of the output 
bits are decoded in finite time, fail to achieve capacity with 
bounded decoding complexity. The accumulation operation 
results in differential encoding of the inner code. The marginal 
uncertainty of the output bits does not decrease. But the joint 
uncertainty decreases substantially, giving us an estimate of 
the inner code bits, and therefore the information bits. 

The arguments above are not tight for the BEC, because 
each edge in the graph needs to be used only once for the BEC. 
However, they hold for general memoryless channels. The 
bounds in Sectionlllll therefore imply that an accumulation-like 
operation is necessary for any sequence of codes to achieve 
capacity with bounded decoding complexity. Accumulation is 
really an example of a convolution code. Therefore, an outer 
convolution code would mean that the bounds in Section |lll] 
do not apply. 

V. DISCUSSIONS AND CONCLUSIONS 

We showed that under message passing decoding, the 
achievable rate of a sparse graph code sequence is bounded 
below capacity if there is a finite success with bounded 
complexity decoding. The bound in Section |lll] is actually 
because a bounded decoding complexity also bounds the 
neighborhood size. Some success in decoding with a bounded 
neighborhood size suggests strong local structures in the code, 
which is detrimental to its performance. 

The bounds imply that an accumulation-type outer code 
would be necessary in order to achieve capacity with bounded 
decoding complexity for general memoryless channels. 
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